12 research outputs found
Test Generation and Dependency Analysis for Web Applications
In web application testing existing model based web test generators derive test paths from a navigation model of the web application, completed with either manually or randomly generated inputs. Test paths extraction and input generation are handled separately, ignoring the fact that generating inputs for test paths is difficult or even impossible if such paths are infeasible. In this thesis, we propose three directions to mitigate the path infeasibility problem. The first direction uses a search based approach defining novel set of genetic operators that support the joint generation of test inputs and feasible test paths. Results show that such search based approach can achieve higher level of model coverage than existing approaches. Secondly, we propose a novel web test generation algorithm that pre-selects the most promising candidate test cases based on their diversity from previously generated tests. Results of our empirical evaluation show that promoting diversity is beneficial not only to a thorough exploration of the web application behaviours, but also to the feasibility of automatically generated test cases. Moreover, the diversity based approach achieves higher coverage of the navigation model significantly faster than crawling based and search based approaches. The third approach we propose uses a web crawler as a test generator. As such, the generated tests are concrete, hence their navigations among the web application states are feasible by construction. However, the crawling trace cannot be easily turned into a minimal test suite that achieves the same coverage due to test dependencies. Indeed, test dependencies are undesirable in the context of regression testing, preventing the adoption of testing optimization techniques that assume tests to be independent. In this thesis, we propose the first approach to detect test dependencies in a given web test suite by leveraging the information available both in the web test code and on the client side of the web application. Results of our empirical validation show that our approach can effectively and efficiently detect test dependencies and it enables dependency aware formulations of test parallelization and test minimization
Boundary State Generation for Testing and Improvement of Autonomous Driving Systems
Recent advances in Deep Neural Networks (DNNs) and sensor technologies are
enabling autonomous driving systems (ADSs) with an ever-increasing level of
autonomy. However, assessing their dependability remains a critical concern.
State-of-the-art ADS testing approaches modify the controllable attributes of a
simulated driving environment until the ADS misbehaves. Such approaches have
two main drawbacks: (1) modifications to the simulated environment might not be
easily transferable to the in-field test setting (e.g., changing the road
shape); (2) environment instances in which the ADS is successful are discarded,
despite the possibility that they could contain hidden driving conditions in
which the ADS may misbehave.
In this paper, we present GenBo (GENerator of BOundary state pairs), a novel
test generator for ADS testing. GenBo mutates the driving conditions of the ego
vehicle (position, velocity and orientation), collected in a failure-free
environment instance, and efficiently generates challenging driving conditions
at the behavior boundary (i.e., where the model starts to misbehave) in the
same environment. We use such boundary conditions to augment the initial
training dataset and retrain the DNN model under test. Our evaluation results
show that the retrained model has up to 16 higher success rate on a separate
set of evaluation tracks with respect to the original DNN model
Testing of Deep Reinforcement Learning Agents with Surrogate Models
Deep Reinforcement Learning (DRL) has received a lot of attention from the
research community in recent years. As the technology moves away from game
playing to practical contexts, such as autonomous vehicles and robotics, it is
crucial to evaluate the quality of DRL agents. In this paper, we propose a
search-based approach to test such agents. Our approach, implemented in a tool
called Indago, trains a classifier on failure and non-failure environment
(i.e., pass) configurations resulting from the DRL training process. The
classifier is used at testing time as a surrogate model for the DRL agent
execution in the environment, predicting the extent to which a given
environment configuration induces a failure of the DRL agent under test. The
failure prediction acts as a fitness function, guiding the generation towards
failure environment configurations, while saving computation time by deferring
the execution of the DRL agent in the environment to those configurations that
are more likely to expose failures. Experimental results show that our
search-based approach finds 50% more failures of the DRL agent than
state-of-the-art techniques. Moreover, such failures are, on average, 78% more
diverse; similarly, the behaviors of the DRL agent induced by failure
configurations are 74% more diverse
Search based path and input data generation for web application testing
Test case generation for web applications aims at ensuring full coverage of the navigation structure. Existing approaches resort to crawling and manual/random input generation, with or without a preliminary construction of the navigation model. However, crawlers might be unable to reach some parts of the web application and random input generation might not receive enough guidance to produce the inputs needed to cover a given path. In this paper, we take advantage of the navigation structure implicitly specified by developers when they write the page objects used for web testing and we define a novel set of genetic operators that support the joint generation of test inputs and feasible navigation paths. On a case study, our tool Subweb was able to achieve higher coverage of the navigation model than crawling based approaches, thanks to its intrinsic ability of generating inputs for feasible paths and of discarding likely infeasible paths
Neural Embeddings for Web Testing
Web test automation techniques employ web crawlers to automatically produce a
web app model that is used for test generation. Existing crawlers rely on
app-specific, threshold-based, algorithms to assess state equivalence. Such
algorithms are hard to tune in the general case and cannot accurately identify
and remove near-duplicate web pages from crawl models. Failing to retrieve an
accurate web app model results in automated test generation solutions that
produce redundant test cases and inadequate test suites that do not cover the
web app functionalities adequately. In this paper, we propose WEBEMBED, a novel
abstraction function based on neural network embeddings and threshold-free
classifiers that can be used to produce accurate web app models during
model-based test generation. Our evaluation on nine web apps shows that
WEBEMBED outperforms state-of-the-art techniques by detecting near-duplicates
more accurately, inferring better web app models that exhibit 22% more
precision, and 24% more recall on average. Consequently, the test suites
generated from these models achieve higher code coverage, with improvements
ranging from 2% to 59% on an app-wise basis and averaging at 23%.Comment: 12 pages; in revisio
Search Based Path and Input Data Generation for Web Application Testing
Test case generation for web applications aims at ensuring full coverage of the navigation structure. Existing approaches resort to crawling and manual/random input generation, with or without a preliminary construction of the navigation model. However, crawlers might be unable to reach some parts of the web application and random input generation might not receive enough guidance to produce the inputs needed to cover a given path. In this paper, we take advantage of the navigation structure implicitly specified by developers when they write the page objects used for web testing and we define a novel set of genetic operators that support the joint generation of test inputs and feasible navigation paths. On a case study, our tool Subweb was able to achieve higher coverage of the navigation model than crawling based approaches, thanks to its intrinsic ability of generating inputs for feasible paths and of discarding likely infeasible paths
Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing
Simulation-based testing represents an important step to ensure the
reliability of autonomous driving software. In practice, when companies rely on
third-party general-purpose simulators, either for in-house or outsourced
testing, the generalizability of testing results to real autonomous vehicles is
at stake.
In this paper, we strengthen simulation-based testing by introducing the
notion of digital siblings, a novel framework in which the AV is tested on
multiple general-purpose simulators, built with different technologies. First,
test cases are automatically generated for each individual simulator. Then,
tests are migrated between simulators, using feature maps to characterize of
the exercised driving conditions. Finally, the joint predicted failure
probability is computed and a failure is reported only in cases of agreement
among the siblings.
We implemented our framework using two open-source simulators and we
empirically compared it against a digital twin of a physical scaled autonomous
vehicle on a large set of test cases. Our study shows that the ensemble failure
predictor by the digital siblings is superior to each individual simulator at
predicting the failures of the digital twin. We discuss several ways in which
our framework can help researchers interested in automated testing of
autonomous driving software
Dependency-Aware Web Test Generation
Web crawlers can perform long running in-depth explorations of a web application, achieving high coverage of the navigational structure. However, a crawling trace cannot be easily turned into a minimal test suite that achieves the same coverage. In fact, when the crawling trace is segmented into test cases, two problems arise: (1) test cases are dependent on each other, therefore they may raise errors when executed in isolation, and (2) test cases are redundant, since the same targets are covered multiple times by different test cases. In this paper, we propose DANTE, a novel web test generator that computes the test dependencies associated with the test cases obtained from a crawling session, and uses them to eliminate redundant tests and produce executable test schedules. DANTE can effectively turn a web crawler into a test case generator that produces minimal test suites, composed only of feasible tests that contribute to achieve the final coverage. Experimental results show that DANTE, on average, (1) reduces the error rate of the test cases obtained by crawling traces from 85% to zero, (2) produces minimized test suites that are 84% smaller than the initial ones, and (3) outperforms two competing crawling-based and model-based techniques in terms of coverage and breakage rate
Diversity-based web test generation
Existing web test generators derive test paths from a navigational model of the web application, completed with either manually or randomly generated input values. However, manual test data selection is costly, while random generation often results in infeasible input sequences, which are rejected by the application under test. Random and search-based generation can achieve the desired level of model coverage only after a large number of test execution at- tempts, each slowed down by the need to interact with the browser during test execution. In this work, we present a novel web test generation algorithm that pre-selects the most promising candidate test cases based on their diversity from previously generated tests. As such, only the test cases that explore diverse behaviours of the application are considered for in-browser execution. We have implemented our approach in a tool called DIG. Our empirical evaluation on six real-world web applications shows that DIG achieves higher coverage and fault detection rates significantly earlier than crawling-based and search-based web test generators
Web test dependency detection
reserved5E2E web test suites are prone to test dependencies due to the heterogeneous multi-tiered nature of modern web apps, which makes it difficult for developers to create isolated program states for each test case. In this paper, we present the first approach for detecting and validating test dependencies present in E2E web test suites. Our approach employs string analysis to extract an approximated set of dependencies from the test code. It then filters potential false dependencies through natural language processing of test names. Finally, it validates all dependencies, and uses a novel recovery algorithm to ensure no true dependencies are missed in the final test dependency graph. Our approach is implemented in a tool called TEDD and evaluated on the test suites of six open-source web apps. Our results show that TEDD can correctly detect and validate test dependencies up to 72% faster than the baseline with the original test ordering in which the graph contains all possible dependencies. The test dependency graphs produced by TEDD enable test execution parallelization, with a speed-up factor of up to 7×.mixedBiagiola M.; Stocco A.; Mesbah A.; Ricca F.; Tonella P.Biagiola, M.; Stocco, A.; Mesbah, A.; Ricca, F.; Tonella, P